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THE  NATURE  OF  UNCERTAINTY 
G.  H.  Fisher 

The  Rand  Corporation,  Santa  Monica,  California 


To  help  set  the  stage  for  our  seminar  the  chairman  has  asked 
me  to  comment  briefly  on  the  nature  of  uncertainty.  Initially,  I 
shall  do  this  by  outlining  some  of  the  dimensions  of  the  problem 
in  general.  Later,  I shall  focus  more  on  the  context  of  resource 
analysis. 

Let  us  start  out  by  making  a distinction  between  risk  and 

•Mt,  ' 

uncertainty.  In  a risky  situation -we- know^ the  objective  probability 
distribution  of  the  random  event  involved  in  the  problem.  In  situa- 
tions  of  uncertainty  we  do  not  know  anything  about  the  objective 
probabilities  involved,  and  hence  the  problem  must  be  dealt  with  in 
terms  of  subjective  considerations  — either  formal  or  informal. 

Sometimes  this  distinction  is  made  in  different  terms;  e.g. , 

"risk"  is  called  statistical  uncertainty,  and  "uncertainty"  is  called 

"kick 

state-of-the-world  or  requirements  uncertainty.  The  relative  im- 

portance of  these  categories  depends  very  much  on  the  context  of  the 
problem  at  hand.  For  Instance  in  most  long  range  planning  problems, 
state-of-the-world  uncertainty  tends  to  dominate  statistical  uncer- 
tainty (risk).  This,  of  course,  has  important  implications  for 
choosing  the  appropriate  concepts  and  methods  for  dealing  with 
uncertainty. 

Another  dimension  of  the  nature  of  uncertainty  pertains  to  what 
may  be  called  the  life  cycle  context.  That  is  to  say,  the  nature  of 


A paper  presented  at  the  Department  of  Defense  Cost  Research 
Symposium,  Alrlle,  Virginia,  November  1973. 

E.g.,  see  Albert  Madansky,  "Uncertainty,"  Chapter  5 in  E.  S. 
Quade  and  W.  I.  Boucher,  Systems  Analysis  and  Policy  Planning  (New 
York:  American  Elsevier  Publishing  Co.,  Inc.,  1968),  pp.  81-96. 

*** 

Examples  of  the  latter  are  uncertainty  about  strategic  con- 
text, technological  uncertainty,  and  uncertainty  about  the  enemy  and 
his  potential  actions  and  reactions. 
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uncertainty  depends  very  much  on  what  phase  of  the  weapon  system  life 

cycle  the  issue  up  for  decision  stems  from;  e.g.: 

Pre-R&D 

Early  R&D 

Late  development 

Production 

Operation. 

Clearly  "state-of-the-world"  uncertainties  dominate  the  pre-R&D 
phase.  This  may  or  may  not  be  so  in  the  case  of  current  operations. 
However,  even  here  if  radically  new  operational  concepts  are  being 
considered,  much  more  than  ’'statistical  uncertainty"  may  be  involved 
in  the  problem. 

Still  another  dimension  of  the  nature  of  uncertainty  is  related 
to  the  level  of  sub-optlralzatlon  being  considered  in  the  decision 
problem  at  hand  — in  sliort , the  "analytical  context."  At  one  ex- 
treme is  the  "total  force"  context;  e.g.,  the  total  USAF  force  struc- 

•k 

ture,  or  the  total  DOD.  At  the  other  extreme  is  a component  of  an 
individual  weapon  system  — e.g.,  the  major  hardware,  or  some  sub- 
component like  the  propulsion  sub-system.  In  between  is  the  indi- 
vidual weapon  system  context.  Obviously,  the  nature  of  the  uncer- 
tainty problem  is  quite  different  depending  upon  which  of  these 
contexts  fits  the  problem  at  hand. 

Let  us  turn  now  to  a brief  discussion  of  the  types  of  methods 
that  may  be  used  to  deal  with  uncertainty.  Since  these  should  be 
familiar  to  most  of  you,  I shall  comment  only  briefly  on  each  one. 
The  following  is  a representative  list  — though  certainly  not  a 
complete  one  — and  the  order  is  not  necessarily  in  accordance  with 
relative  importance: 

1.  Monte  Carlo:  a technique  for  reckoning  with  statistical 

fluctuations  by  drawing  random  samples  from  a carefully 

kk 

determined  distribution.  It  is  most  useful  in  dealing 

k 

Another  'total  force"  context  — perhaps  more  interesting  than 
the  ones  mentioned  above  — is  the  total  force  in  some  mission  area. 
An  example  would  be  the  total  strategic  force  made  up  of  Army,  Navy, 
and  USAF  weapon  systems  and  related  support. 

kk 

For  a good  brief  discussion  of  the  technique,  see  E.  S.  Quade 
(ed.).  Analysis  for  Military  Decision  (Chicago:  Rand  McNally  & Co., 

1964),  pp.  240-243. 
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ti/ith  situations  of  risk  or  statistical  uncertainty.  How- 
ever, it  is  sometimes  used  in  otlier  situations  where  sub- 
jective probability  distributions  are  substituted  for 
* 

objective  ones. 

2.  A fortiori  analysis;  in  comparative  studies  the  systems 
analyst  may  resolve  all  major  uncertainties  in  favor  of 
the  alternatives  to  the  intuitively  preferred  solution. 

If  the  latter  holds  up  well  in  such  a comparison,  the 
analyst  has  a powerful  argument  in  favor  of  the  intuitively 
preferred  alternative. 

3.  Sensitivity  analysis:  in  various  types  of  analytical 

studies,  rather  than  merely  settling  for  "expected  values," 
the  analysts  may  vary  the  values  of  key  uncertain  para- 
meters or  assumptions  over  their  relevant  ranges  to  see 
what  the  impact  is  on  final  outcomes  (e.g.,  ranking  of 
alternatives) . This  can  be  very  useful  to  decisionmakers 
in  deciding  about  which  uncertainties  are  really  Important 
and  hence  are  candidates  for  further  deliberation  — e.g., 
seeking  ways  to  reduce  or  hedge  against  these  uncertainties. 

4.  A 'range  of  estimates":  in  a sense  this  is  a special  case 

of  (3)  above.  Here,  in  addition  to  an  "expected  value" 
case,  several  other  outcomes  are  calculated  corresponding 
to  alternative  assumptions  about  how  key  uncertainties  are 
resolved.  One  mode  that  is  often  used  is  the  range  of 


E.g.,  see  P.  F.  Dleneman,  Estimating  Cost  Uncertainty  Using 
Monte  Carlo  Techniques , RM-4854-PR  (Santa  Monica,  Calif.:  The  Rand 

Corporation,  January  1966). 

A A 

For  a discussion  of  cost  sensitivity  analysis,  see  G.  H.  Fisher, 
Cost  Considerations  in  Systems  Analysis  (New  York:  American  Elsevier 

Co.,  Inc.,  1971),  pp.  215-217. 

A 

A special  case  of  sensitivity  analysis  is  the  so-called  "break- 
even" analysis.  Here,  in  the  case  of  a very  key  parameter  about 
which  we  are  uncertain,  the  analysts  can  calculate  that  value  of 
the  parameter  which  must  be  assumed  in  order  to  change  the  ranking 
of  the  most  Important  alternatives  being  considered. 

4. 

For  this  technique  to  be  useful  to  decisionmakers,  the  alter- 
native assumptions  must  be  set  forth  explicitly. 
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estimates  characterized  by  "high,  ' "medium,"  and  "low." 

5.  Supplemental  discounting!  analysts  very  often  use  the  pro- 
cess of  discounting  future  cost  and  benefit  streams  to 
equalize  them  with  respect  to  time  preference.  Advocates 
of  supplemental  discounting  suggest  that  an  additional 

rate  be  applied  to  allow  for  uncertainties  about  future  i 

technology,  the  enemy  threat,  etc.  Most  analysts  today 
prefer  to  distinguish  between  discounting  for  time  prefer- 
ence and  for  uncertainty;  and  in  general  they  prefer  to 
treat  uncertainty  as  an  explicit  problem  in  the  systems 
analysis  process,  and  not  through  the  use  of  an  aggregate 
supplemental  discount  factor. 

6.  Adjustment  factors:  the  use  of  "adjustment  factors"  to 

allow  for  requirements  uncertainty  has  been  advocated 

primarily  for  application  in  cost  analyses  of  future  major 

* 

equipment  proposals.  It  is  argued  that  because  of  uncer- 
tainties about  technology  and  state-of-the-art  advances, 
etc.,  cost  estimates  made  during  the  early  stages  of  the 
hardware  development  cycle  should  be  adjusted  upward  by 
using  "average  factor  increases"  derived  statistically 
from  historical  records. 

7.  Special  studies:  very  often  systems  analysts  can  be  of 

considerable  help  to  the  decisionmakers  by  conducting 
special  studies  to  explore  the  benefits  and  costs  of  various 
strategies  for  hedging  against  uncertainty  in  some  particu- 
lar area,  to  examine  the  benefits  and  costs  of  "buying  time" 
in  order  to  gain  more  Information  and  hence  reduce  uncer- 
tainty, etc.  ' 

1 

So  much  for  concepts  and  methods  for  dealing  with  uncertainty. 

Let  us  now  change  gears  slightly  and  consider  some  of  the  implications 

*L.g.,  see  Robert  Summers,  Cost  Estimates  as  Predictors  of 
Actual  Weapon  Costs:  A Study  of  M-^jor  Hardware  Articles,  RM-3061-PR 

(abridged)  (Santa  Monica,  Calif.:  The  Rand  Corporation,  March  1965). 
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of  tiie  preceding  discussion.  We  shall  start  by  considering  the  follow- 
ing questioir:  Given  the  dimensions  of  the  nature  of  uncertainty  out- 

lined above,  and  given  the  types  of  concepts  and  methods  for  dealing 
with  uncertainty,  what  are  the  Implications  for  the  development  of 
analytical  tools  and  techniques? 

Clearly,  the  list  of  implications  is  a long  one.  However, 
because  of  time  constraints,  I can  only  present  a few  examples  to 
serve  as  illustrations.  I am  sure  that  most  of  you  in  the  audience 
can  easily  extend  the  list  in  your  own  minds. 

1.  If  we  are  systems  cost  analysts  serving  the  "pre-R&D" 
planning  process,  we  no  doubt  will  have  to  help  the  decision- 
makers grapple  with  state-of-the-world  ("requirements") 
uncertainty.  One  of  the  main  ways  to  do  this  is  to  engage 

in  a considerable  amount  of  ingenious  sensitivity  testing. 

But  in  order  to  do  really  effective  cost  sensitivity  analy- 

* 

sis,  we  have  to  have  good  parametric  cost  models  — models 
for  individual  systems,  sub-systems,  and  force  mixes  (e.g., 
an  inter-service  strategic  force  mix  cost  model).  Wliile 
much  progress  has  been  made  in  recent  years  in  developing 
good  parametric  cost  models,  much  more  remains  to  be  done.  I 
would  say  that  the  effectiveness  of  our  ability  to  deal  sub- 
stantively with  problems  of  uncertainty  is  directly  related 

to  the  availability  of  a wide  range  of  well-designed  parametric 
** 

cost  models. 

2.  If  we  are  dealing  with  an  individual  total  weapon  system 
context,  and  we  are  required  to  treat  the  problem  of  sta- 
tistical uncertainty  (or  risk)  in  the  cost  estimates,  the 


A parametric,  cost  model  is  one  that  is  "open-ended"  with  respect 
to  key  cost-generating  explanatory  variables  such  as  major  equipment 
performance  characteristics,  operational  concepts,  deployment,  force 
size,  etc.  For  an  example,  see  the  hard  point  defense  cost  model  pre- 
sented in  Chapter  7 of  my  Cost  Considerations  in  Systems  Analysis. 

Parametric  cost  models  are  also  very  useful  in  doing  a fortiori 
analysis,  break-even  analysis,  etc. 

irk 

However,  as  I shall  indicate  shortly,  parametric  models  cannot 
do  the  entire  Job.  Important  supplementary  measures  must  usually  be 
employed. 


Monte  Carlo  technique  may  be  appropriate.  However,  most 

past  and  current  applications  have  certain  deficiencies. 

One  stems  from  the  fact  that  in  many  instances  the  individual 

elements  of  weapon  system  cost  are  assumed  to  be  independent, 

when  in  reality  they  are  not.  Thus,  we  need  ways  to  take 

* 

into  account  the  interdependencies.  More  important,  perhaps, 
is  that  in  most  applications  the  cost  analysts  use  their  in- 
tuition to  make  assumptions  about  the  probability  distributions 
for  the  various  elements  in  total  system  cost.  In  many  cases 
it  is  not  clear  that  the  analysts  have  sufficient  substantive 
knowledge  in  specific  areas  to  make  meaningful  subjective 
probability  specifications;  and  as  a result,  some  critics 
question  tiie  value  of  a distribution  of  weapon  system  total 
cost  derived  by  Monte  Carlo  procedures  as  practiced  to  date. 

3.  As  a final  example  let  us  take  the  context  of  hardware  cost 
analysis  where  we  have  to  make  cost  estimates  for  a new  ad- 
vanced major  equipment  item  that  would  be  operational  many  years 
from  now.  In  problems  like  this  we  are  usually  faced  with 
substantial  technological  and  manufacturing  state-of-the-art 
uncertainties.  Wliile  statistically-derived  parametric  cost 
models  can  help  considerably  in  dealing  with  these  problems, 
they  can  only  take  us  so  far.  One  of  the  main  reasons  is  that 
they  are  derived  from  an  historical  data  base  that  does  not 
include  the  advanced  equj.pment  item  under  consideration. 

Thus,  the  use  of  parametric  cost  models  has  to  be  supplemented 

by  the  results  of  special  studies.  For  example,  the  analysts 

may  liave  to  go  to  various  government  and  contractor  facilities 

where  experiments  are  being  conducted  on  new  materials,  manu-  ^ I 

factoring  techniques,  prototype  sub-systems,  and  the  like.  ' 

j 

Information  gained  from  such  field  work,  even  though  some  of  it  i 

may  be  qualitative,  can  be  of  great  help  in  establishing  mean-  j 

ingful  ranges  of  cost  estimates  for  the  advanced  hardware  item 
under  consideration. 

* 

I understand  that  some  work  along  this  line  is  currently  under 

way. 

j 

I 
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As  a final  topic,  I would  like  to  consider  briefly  a few  points 
pertaining  to  the  implications  of  the  preceding  discussion  for  DOD 
policy : 

1.  Can  we  someliow  maintain  ranges  of  cost  estimates  in  the 
”D0D  books,"  rather  than  just  point  estimates?  Probably 
not,  when  the  context  is  the  conventional  budget.  But  in 
the  planning  and  prograiunlng  realms,  having  a range  of 
estimates  for  DOD  in-house  use  would  seem  feasible  and 
desirable  — especially  for  Important  major  parts  of  the 

■k 

defense  program. 

2.  Related  to  (1)  is  the  matter  of  "advocacy  cost  estimates." 

I have  been  led  to  believe  that  because  of  new  DSARC  pro- 
cedures this  is  no  longer  the  problem  it  once  was.  But  to 
the  extent  that  advocacy  estimates  are  still  developed  and 
used,  DOD  policy  should  require  that  for  "in-house"  DOD 
use  other  (presumably  more  realistic)  estimates  be  avail- 
able "in  the  system,"  so  that  decisionmakers  will  have 
alternative  sources  of  information  rather  than  merely  the 
advocacy  source. 

3.  We  must  continuously  watch  the  allocation  of  DOD's  ana- 
lytical resources  between  the  development  of  formal  quanti- 
tative methods  for  dealing  with  uncertainty  (usually  sta- 
tistical uncertainty)  on  the  one  hand,  and  the  conduct  of 
ad  hoc  special  studies  to  help  deal  with  state-of-the-world 


Here,  proper  documentation  is  important.  It  is  not  sufficient 
to  merely  have  a range  of  numbers  "in  the  books."  In  addition,  each 
case  must  be  supplemented  with  explicit  descriptions  of  the  key 
assumptions  underlying  each  particular  estimate  in  the  range.  Purely 
quantitative  treatments  are  not  enough. 

kk 

Use  of  advocacy  estimates  inevitably  leads  to  unreal  and  un- 
necessary "uncertainty."  Because  use  of  advocacy  estimates  almost 
by  definition  means  there  will  be  substantial  (often  huge)  "cost 
growth,"  some  people  observing  the  process  might  conclude  that  the 
early  estimates  were  made  under  conditions  of  more  real  uncertainty 
than  was  actually  necessary.  Alternative  sets  of  early  estimates 
could  have  exposed  the  advocacy  estimates  for  what  they  in  fact  were. 
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("requirements")  uncertainty  on  the  other.  While  both 
types  of  effort  are  needed  and  should  be  pursued,  I sometimes 
feel  that  more  effort  (including  travel  budget)  should  be 
allocated  to  the  special  study  efforts.  This  is  especially 
true  if  uncertainty  is  to  be  dealt  with  adequately  in  "design- 
to-cost"  deliberations.  Here,  as  in  most  cases,  parametric 
methods  derived  from  historical  data  bases  will  not  do  the 
total  job.  They  must  be  supplemented  by  the  results  of 
special  studies. 


As  I indicated  previously  such  special  study  efforts  can  be 
directed  toward  assisting  decisionmakers  in  considering  the  costs 
and  benefits  of  buying  more  information,  of  keeping  options  open,  of 
pursuing  parallel  exploratory  development  efforts,  etc. 


